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ABSTRACT 



A research project identified and prototyped critical 
elements of an automated development system for interactive courseware (ICW) 
for job training. The prototype system incorporated a multimedia database 
that contained all completed, planned, and partially completed media 
elements. A web-based approach was taken to overcome the problems of platform 
dependence. The system consisted of the following elements: an electronic 
storyboard that allowed designers to enter data elements as they performed 
preproduction design and development activities; a media production tool that 
provided access to and reports from the multimedia database; and a project 
management tool that provided information to ICW development managers . The 
system definition involved the following steps: (1) capture the concept of 

operations; (2) document a concise set of requirements; (3) develop a 
preliminary database architecture; (4) select appropriate system hardware; 

(5) design the network architecture; and (6) select commercial off-the-shelf 
software. Preliminary use of the system identified additional features that 
would make the system more useful for ICW development such as a distributed 
media repository to replace the current database structure; a full-text 
search capability; an encryption capability; and teaming tools. A comparison 
of the time required to develop one hour of ICW using the automated approach 
and the traditional approach showed the time was reduced by approximately 30 
percent when the automated system was used. (YLB) 
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Since the early 1970s, the computer-based training industry has grown to the point where most American 
government agencies and private organizations, and a significant number of international organizations, 
implement some type of interactive courseware (ICW) in their training programs. Some organizations 
rely solely on ICW to deliver training, while others combine ICW with other delivery methods, such as 
C4 classroom instruction, on-the-job training, and simulator-based training. 
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Historically, the process of developing ICW has been viewed as complicated and risky. There are as 
many documented cases of cost overruns and poor-quality products as there are successful programs. 
However, even with the risks and high costs associated with ICW, it has emerged as the most popular 
training delivery system within the last 30 years. With more powerful computing technologies now 
available at lower cost, and the emergence of digitial multimedia, the popularity of ICW is increasing. 



It has long been accepted by those who develop multimedia ICW that it is necessary to automate critical 
aspects of the ICW development process. There is less agreement on which aspects should be 
automated and the approach that should be taken to automation. This paper will describe a research 
project in which critical elements of an automated development system were identified and prototyped. 

The prototype system incorporates a multimedia database which contains all completed, planned, and 
partially completed media elements. A Web-based approach was taken to overcome the problems of 
platform dependence. The system consists of an electronic storyboard tool which allows designers to 
enter data elements as they perform preproduction design and development activities; a media 
production tool which provides access to and reports from the multimedia database; and a project 
management tool which provides information to ICW development managers. 



A comparison of ICW development using the traditional approach and the automated approach with the 
prototype system was performed. The result was a reduction in labor hours of approximately 30 percent 
using the automated approach. 
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BACKGROUND: THE NEED FOR AN 
AUTOMATED SYSTEM FOR INTERACTIVE 
COURSEWARE DEVELOPMENT 

Interactive courseware (ICW) is an instructional 
medium in which computer hardware and software 
are used to deliver instruction and test students on 
their ability to understand content and perform 
tasks. Due to the rapid development of computer 
technology, particularly in multimedia such as 
digital audio, video, and graphics, ICW has 
become the preferred instructional delivery 
system for many training programs. The 
Department of Defense has embraced ICW as a 
primary instructional delivery system for new 
weapons systems, and many older classroom- 
based systems are being converted to ICW. It has 
long been accepted by those who have dealt with 
the complexity of developing multimedia ICW that 
it is necessary to find a way to automate the most 
critical aspects of the ICW development process. 
The following paragraphs describe some of the 
problems inherent in the traditional method of 
developing ICW. 

Development and Review of Storyboards 

Storyboards generally contain descriptions of the 
text and visuals, programming notes, text for 
audio narration, correct student responses, and 
feedback in response to student actions. 
Development of paper-based storyboards has 
always been a time-consuming process. Each 
time a revision is made, a complete set of 
storyboards are usually reprinted and provided to 
the reviewers. Careful review of storyboards by 
the subject matter experts and clients is necessary 
before money is expended to develop the media 
and program the ICW. This is a major weakness 
in the process since many clients find it difficult or 
impossible to visualize the “look and feel” of a 
lesson from paper-based storyboards, especially 
when 3D animation or motion video is described. 
Often it is difficult to convince clients to provide 
careful technical review of paper-based 
storyboards. Only when the ICW is programmed 
and the clients can sit at the computer and see a 



lesson, do they carefully review the content for 
technical accuracy. It is at this point, after 
programming has been done, that changes can be 
expensive. 

Storage and Retrieval of Media Elements 

Traditionally, media elements (photographs, video 
segments, 2D and 3D graphics, audio) are 
digitized and inserted into the ICW as the final 
step in the process. When media elements are 
completed for one lesson, they are not easily 
accessed by other members of the development 
team. For some projects, a large number of visual 
media elements exist at the start of a project, such 
as one where ICW is being developed for an 
existing aircraft system. However, instructional 
designers do not have time to manually search 
through boxes of photographs and schematics to 
find the one photograph that might work on the 
storyboard frame they are designing. 

Management of the Development Process 

Managers frequently ask designers to provide 
data on “percentage complete.” Calculating this 
number is a tedious and inexact science, 
especially on large ICW projects where hundreds 
of ICW lessons are being developed. A method is 
needed to easily determine how much work has 
actually been completed, such as how many 
storyboards have been designed and reviewed by 
subject matter experts, how many media elements 
have been produced and digitized, and how many 
lessons have been reviewed. 

DESCRIPTION OF THE RESEARCH 
APPROACH 

Our experience has shown that the aspects of the 
ICW development process that would most benefit 
from automation are development and review of 
storyboards, storage and retrieval of media 
elements (graphics, audio, animation, video 
segments), and management of the development 
process. We believe that a system for automating 
the ICW development process should provide the 



ability to develop high-quality ICW with fewer 
development hours and less experienced 

personnel, and the ability to produce an improved 
product by facilitating early and clear 

communication between all persons involved in 
the design and development of the ICW. Further, 
we believe that this system should facilitate 
improv'ed i ■ icn icigciUcni cirio proouciion of 

courseware by providing communication and data- 
sharing tools to project managers, designers, 
clients, subject matter experts, and other 
production support staff such as photographers, 
video production teams, and narrators. 

Research was needed to determine whether an 
automated system could be built which would 
provide these capabilities. Our research effort 
consisted of defining the system and building a 
prototype. We determined that the most important 
feature of this prototype system would be a 
multimedia database for storage and retrieval of 
all types of media elements. This multimedia 
database would need interfaces for various users 
(members of the ICW development team) to enter 
and extract information. A Web-based approach 
was selected to overcome problems related to 
platform dependence. The system uses standard 
Internet protocols. Users need only a standard 
Internet Web browser such as Netscape Navigator 
or Microsoft Internet Explorer. The system will 
support users with Windows, Windows 95, 
Windows NT, OS/2, Macintosh, or UNIX operating 
systems. 

DESCRIPTION OF THE PROTOTYPE SYSTEM 

The system consists of an electronic storyboard 
tool, a media production tool, and a project 
management tool which are “front ends” to a 
multimedia database. 

The electronic storyboard tool allows designers to 
specify media elements and instructional flow, 
browse a database to find existing media 
elements, and receive review comments for 
updates. The electronic storyboard tool also 
provides reviewers, including clients, subject 
matter experts, and managers a means to review 
and approve lessons electronically at varying 
stages of completeness. In the final review, they 
would be able to see or hear the media elements 
related to each storyboard frame. 

The media production tool provides reports to be 
used by those who develop media elements 



(videographers, graphic artists, animators, 
photographers, narrators) which would replace 
manually developed shot lists, narrative scripts, 
etc. The media production tool provides an 
interface for entering digitized audio and video 
elements into the system, as well as information 
such as key words, to help others find the media 
elements after they have been entered into the 
database. 

The project management tool takes input from the 
storyboard and media production tools and 
provides reports to managers to help them 
determine project status. 

HOW THE ICW TEAM MEMBERS USE THE 
AUTOMATED SYSTEM 

The easiest way to understand the concept of the 
automated system is from the perspective of the 
various members of the development team: 
instructional designers, subject matter experts, 
media producers (videographers, graphic artists, 
photographers, programmers/software designers, 
narrators), media database producers, managers, 
and clients. Figure 1 illustrates how the ICW team 
members use the system. 

ICW Designer 

The ICW designer uses the system to create a 
structure for the ICW consisting of courses, units, 
and lessons. For each lesson, the designer 
completes as many storyboard frames as 
necessary. For each frame, the designer enters 
visuals, audio, text, programming notes, and 
correct responses. Visuals may include 2D and 3D 
graphics, still photos, animation, and video 
segments (digitized). 

The designer can enter visuals and audio into the 
storyboard in the following ways: 

• “Request” media elements (video segments, 
graphics, audio narration) by entering 
descriptions of audio and visual elements. 
These “requests” are turned into shot lists, 
narrative scripts, etc., for the media production 
staff. The designer can also make rough 
drawings of visual elements using an electronic 
drawing pad, or scan in pictures similar to 
those desired. Once the media elements are 
produced and digitized, they can be reviewed 
on the storyboard page. 
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Figure 1. ICW Development Using the Automated System 
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• “Find” media elements by browsing in a 
repository of visual or audio elements, 
then refine the search by looking at the 
visual element or listening to the audio 
element. \A/hen a media element is 
selected, it becomes attached to the 
storyboard and can be reviewed on the 
storyboard page. In some cases, visual 
and/or audio elements will be loaded into 
the multimedia database at the start of the 
project. For example, if the subject is 
maintenance training, it is very likely that 
before the designers begin developing 
storyboards, visual elements of all the 
equipment components would be entered 
into the multimedia database. Much time 
would be saved by not having to describe 
objects when the actual photograph or 
video clip can be entered. 

In some cases it may be advantageous to obtain, 
digitize, and enter visual media elements into the 
system before any storyboarding is done. In this 
way, individual designers can see what the 
equipment actually looks like rather than having to 
refer to poor-quality visuals in technical manuals 
as they develop a lesson. They can immediately 
“attach” a media element to the storyboard frame 
if the visual is appropriate, or they can annotate 
the visual, explaining what changes need to be 
made to be useful for the storyboard frame. 

Reviewers 

Reviewers of ICW may be customers, subject 
matter experts, and managers. In some cases the 
project may call for subject matter experts and 
clients to review the electronic storyboards twice. 
The first review of the electronic storyboards is 
accomplished when the media elements are 
descriptions or rough drawings. In this way, they 
can ensure that the correct media elements will be 
produced. The second review is completed when 
the digitized media elements are “attached” to 
each electronic storyboard page. Reviewers can 
see the media elements before the whole lesson 
is actually programmed. With V\feb-based access, 
reviewers can review the media elements as they 
become available. The reviewers can add 
comments to any storyboard frame. 

ICW Program Manager 

The ICW program manager receives reports on 
percentage of work completed using the following 



parameters: percentage of total storyboards 

completed by the designers, lessons completed, 
media elements completed, and approvals 
received. 

Media Producer/Media Database Producer 

The producer of the visual and audio media 
elements uses reports generated by the system, 
such as shot lists, narrative scripts, and rough 
drawings, to produce the media elements. When 
the media elements are completed, the media 
database producer digitizes them and enters them 
in the database/media repository. Keyword 
descriptions are entered to help designers find 
media elements that are already in the system. 

ICW Programmer 

The multimedia ICW programmer, who uses 
either special-purpose authoring tools or general- 
purpose programming languages to author ICW, 
can print programming notes or open the system 
in a window, while programming the lesson. The 
programmer can import digitized media elements 
from the multimedia database into the lesson 
since all media elements are saved as files. 

HOW THE AUTOMATED SYSTEM WAS 
DEVELOPED 

The system definition involved the following steps: 
(1) capture the concept of operations, (2) 
document a concise set of requirements, (3) 
develop a preliminary database architecture, (4) 
select appropriate system hardware, (5) design 
the network architecture, and (6) select COTS 
software. Of primary importance during these 
tasks was the determination of the proper 
combination of hardware, software and network 
architecture for the system. The decision was 
made to use open, Internet-based standards 
instead of proprietary, Windows-only solutions. 
This decision to use “open standards” is based on 
the belief that the raw power of Internet 
technologies such as Web servers, Web 
browsers, Java and client/server databases more 
than makes up for the problems to be expected 
using such emerging technologies. 

The system design may be summarized as a 
partitioned client/server JavaScript system. The 
system concentrates almost all of the computation 
and database work on the server. The small 
amount of code that must run on the client’s 



machines is loaded into the user’s Web browser 
by the system while it is running. This happens 
transparently to system users. The system is able 
to maintain its high level of platform 
independence because the part of the application 
running on a client computer is written in 
JavaScript, which is supported by most major 
Web browsers under most operating systems. 

Network Architecture 

The network architecture is a wide area network 
(WAN) which uses standard Internet protocols. 
This allows access to the system from remote 
sites connected to the Internet. Connections from 
networks other than the assigned members of the 
development team are disallowed unless 
specifically enabled. Internet-based networking 
allows the system to easily support remote users 
and leaves the door open for supporting multiple 
types of client systems in the future. In many 
image-intensive systems, Macintosh or UNIX 
computers are used, due to their ease of use 
(Macintosh) and power for editing graphic images 
and digital video content (UNIX). Another reason 
for the selection of an Internet-based network 
architecture was that it could easily mature into a 
distributed data repository instead of a centralized 
data repository. The database information needed 
to track and manage courseware development 
efforts will always reside in a single database 
system somewhere on the network, but the actual 
media files (images, digital video, etc.) might be 
stored in a distributed manner on the other 
computers being used to produce an ICW product. 
The availability of free or very inexpensive Web 
servers for most PC platforms makes this a real 
possibility. The major benefit of this approach is 
that allowing a user to store media files on a local 
computer provides improved performance and 
lower network traffic. 

Software Used to Develop the System 

The system is written in an enhanced version of 
the JavaScript programming language with 
commercial off-the-shelf (COTS) “add-ons" to 
provide hooks to a V\feb server, hooks into one or 
more database subsystems, and hooks into the 
operating system, Web server, and application 
security subsystems. The system uses the 
following commercial software products: 



• Netscape “FastTrack” Web server, Microsoft 
“FTP Server" (included with NT) 

• Borland International “Intrabuilder" develop- 
ment system for client/server JavaScript 
development 

• Borland International “Borland Database 
Engine" to provide server-side SQL 
database support (included with Intrabuilder) 

Database Architecture 

The system is built on a shared multimedia 
database that underlies special-purpose user- 
interface screens that support users in the 
paradigms in which they are used to working, such 
as the electronic storyboard for instructional 
designers, reviewers, and programmers; reports 
for media producers containing shot lists, 
narrative scripts, etc.; management reports; and 
forms for entry of information into the multimedia 
database. 

The database architecture selected for the system 
is a hybrid database/file repository scheme in 
which database records reference actual media 
element files that reside in data repositories. A 
data repository is simply a specification of a 
particular directory. Currently, all system data 
repositories reside on the system server. Support 
for data repositories on other computers on the 
local network is one of many planned 
enhancements to the system. 

The system uses local Paradox or Dbase file 
formats, which provides good performance without 
the need to license additional database software. 
It is unnecessary to use a conventional relational 
database system because the Borland Database 
Engine supports most major client/server 
database systems, including Microsoft SQL 
server, Oracle, and Borland InterBase. Local 
database file formats such as Access, Paradox, 
and Dbase are also supported, as is access to 
ODBC databases from the server. By using 
standard SQL queries in the system source code, 
the system could very easily be adapted to 
support any of the database systems above. For a 
larger application, a dedicated database server 
running Oracle or MS-SQL server could be added 
to the network and the same SQL statements 
would still work. For even larger user communities 
the system software may be distributed across 
multiple system servers. 



Security 

Security in the automated system is handled by 
user IDs and passwords combined with the use of 
Intranet restrictions. The Intranet restrictions mean 
that only IP addresses within the company domain 
are allowed to connect to the system server, and 
that the local network that the system runs on is 
protected against forged IP packets that claim to 
be part of the local network but that actually 
originate from outside. Because the system is built 
on top of Web server technology, it is possible to 
use the secure Web server technology being 
offered by both Netscape and Microsoft. An 
enhancement to the system is planned which will 
provide an additional layer of security, based on 
digital certificate encryption technology that 
effectively scrambles all communication between 
the system V\feb server and a user’s Web browser. 

Hardware/Software Required to Use the 
Automated System 

The system does not require that any special 
software be installed on the user’s computer. 
Users access the system using a standard Internet 
V\feb browser such as Netscape Navigator or 
Microsoft Internet Explorer. Users with Windows, 
Windows 95, Windows NT, OS/2, Macintosh, or 
UNIX computers can access and use the system. 
All system user-interface “forms,” such as the 
electronic storyboard tool, present themselves as 
V\feb pages to system users. These forms use 
standard Hypertext Markup Language (HTML)- 
based V\feb user interface mechanisms such as 
buttons to click on, text entry areas, and drop- 
down lists. Users navigate within the system by 
simply clicking on buttons to open and close user- 
interface forms. The server requires a PC with 
Microsoft Windows NT (workstation or server). 

ENHANCING THE CAPABILITIES OF THE 
AUTOMATED SYSTEM 

Preliminary use of the system resulted in 
identification of the following additional features 
which will make the system more useful for ICW 
development, especially large ICW development 
efforts. 

• A distributed media repository to replace 
the current database structure. This will 
allow designers to browse the entire 
media repository, while at the same time 
they will have rapid access to the images 



and sound files they are most likely to 
use. For example, images related to 
maintenance would exist on the server of 
a designer of maintenance training, while 
images related to operation would exist on 
the server of a designer of operator 
training. However, both designers would 
be able to access all images in the media 
repository. 

• A full-text search capability to facilitate 
designers who are searching in the media 
repository for desired media elements. 

• An encryption capability to allow safe 
access to the system from geographically 
disbursed customers, managers, and 
subject matter experts to review and sign 
off on the ICW at various stages in its 
development. 

• A capability to store and play executables 
that will allow the system to treat 
simulations and animations, which are 
written in standard programming 
languages, like any other digital media 
element. 

• Tools that will make it easier to load 
completed media elements into the media 
repository and which will facilitate 
generation and copying of media 
repositories. 

• Teaming tools that will further facilitate 
the sharing of information, such as a 
whiteboard and internal e-mail. 

• Additional user interfaces and reports 
generated by the system, as well as 
enhancements to existing user interfaces 
and reports, which will be developed, 
based on recommendations from 
managers, media professionals, instruc- 
tional designers, and clients using the 
system. 

• An icon-based system for creating and 
accessing courses, lessons, and frames, 
like those used in various authoring 
systems. 

Further investigation will be conducted into the 
feasibility of developing the capability to port the 
overall course structure defined in the system 



software into one or more courseware authoring 
tools and to provide direct access to the system 
database tables from within one or more 
commercial ICW authoring systems. It is believed 
that these features will save additional 
programming time as well as ensure that the 
desired structure is properly captured in the final 
ICW product. 

TIME AND COST SAVINGS USING THE 
AUTOMATED SYSTEM 

A comparison of the time required to develop one 
hour of ICW using the automated approach and 
the traditional approach was performed. The time 
data for the traditional approach is based on 
historical data from a recent project. The time 
data for the automated approach is based on a 
development effort in which the automated 
system was used. In this example, all types of 
media were produced for the lesson. The ICW 
developed is PC-based using software-only digital 
video and COTS authoring graphics/software. The 
times listed in Table 1 represent the combined 
total labor hours for all staff members participating 
in the project for each task. A description of the 
reason for the time savings attributed to using the 
automated system is shown in italics. 

In this example, the time was reduced by 
approximately 30 percent when the automated 
system was used. We expect that the time ratios 



between traditional and automated approaches 
will widen as more hours of ICW are produced. 
For example, if 200 hours of ICW are being 
developed, the time to prepare, deliver, receive 
comments, revise both the storyboard and the 
actual lessons, and manage the overall effort will 
increase with the traditional approach and 
decrease with the automated approach. It is 
estimated that the cost and time for 200 hours 
could be reduced by up to 50 percent if the 
automated system is used. 
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Table 1. ICW Project Development Using Traditional and Automated Approaches 



A j. ^ if, . ^ m • 


APPROACHES - - ^ 


lew TASKS 


TRADITIONAL 


AUTOMATED 


1 . Analyze training content 


24 


24 


2. Develop instructional “treatment” plan 


24 


24 


5. Prepare storyboard 

The electronic storyboard makes it easier to enter information 
and make changes as the designer is "evolving" the lesson. 
No more "cut and paste, " "white out, " or Xeroxing. 


40 


20 


4. Deliver storyboard 


4 


0 


Traditionally, four hours is the time required to Xerox, write 
the delivery letter, package, and mail. Although not labor 
hours, additional time is also required for the customer to 
receive the storyboards through the postal service. 






5. Receive sponsor review comments on storyboard 


2 


0 


Usually 5 days are required to receive comments via postal 
service; 2 hours are required to unpack and make clerical 
configuration control entries. 






6. Revise storyboard 


8 


2 


In paper format, revisions require fixing graphics, editing text. 
Xeroxing, re-mailing, etc. 






7. Receive approval of storyboard 


2 


0 


With the traditional approach, sponsors may take months to 
send approval letters. With the automated system, the 
reviewer can review and approve, using a secure code. 






8. Produce media 






• Record audio 


24 


4 


• Develop graphics and animation 


40 


30 


• Record motion video 


8 


8 


• Take still photographs 


8 


8 


Redundancy of effort (e.g., more than one person producing 
the same media element) is eliminated. Audio exists in files in 
the system, which makes it easier for the developer to 
manipulate. 






9. Program lesson using authoring system 

Programming effort is reduced because all media elements 
are stored in the resource database. 


40 


20 


10. Deliver lesson 


2 


2 


1 1 . Receive sponsor comments 


8 


8 


12. Revise lesson 


8 


8 


13. Conduct evaluation using students 


8 


8 


14. Revise lesson 


2 


2 


15. Deliver final lesson 


2 


2 


16. Overall management 


4 


0.5 


With the automated system, management reports and on- 
line review capability decrease management time 






TOTAL 


258 


170.5 
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